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Abstract 

Many organisms live in populations structured by space and by class, exhibit plastic responses to their 
social partners, and are subject to non-additive ecological and fitness effects. Social evolution theory has 
long recognized that all of these factors can lead to different selection pressures but has only recendy 
attempted to synthesize how these factors interact. Using models for both discrete and continuous phe- 
notypes, we show that analyzing these factors in a consistent framework reveals that they interact with 
one another in ways previously overlooked. Specifically, behavioral responses (reciprocity), genetic re- 
latedness, and synergy interact in non-trivial ways that cannot be easily captured by simple summary 
indices of assortment. We demonstrate the importance of these interactions by showing how they have 
been neglected in previous synthetic models of social behavior both within and between species. These 
interactions also affect the level of behavioral responses that can evolve in the long run; proximate bi- 
ological mechanisms are evolutionarily stable when they generate enough responsiveness relative to the 
level of responsiveness that exacdy balances the ecological costs and benefits. Given the richness of social 
behavior across taxa, these interactions should be a boon for empirical research as they are likely crucial 
for describing the complex relationship linking ecology, demography, and social behavior. 
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1. Introduction 

Explaining the evolution of social behaviors has been a goal of evolutionary theory going back to Dar- 



win's time. The modern theory for the origin of social behaviors started with the seminal work of Hamilton 
( ]l964a| fl, which showed that genetic relatedness has a profound influence on evolutionary dynamics. Later 



work added other significant factors, starting with conditional and responsive behaviors (]Trivers| , |1971g |Axel- 



rod and Hamilton! , |1981| ) and extending to non-additive fitness interactions ( jQueller] , |1984] , |1985[ ). Theorists 
analyzed many models that apply and extend these mechanisms (e.g., extending reciprocity to include indi- 



rect responses, [Nowak and Sigmund| , |1998[ ). At the same time, empirical researchers tested the role of various 



mechanisms developed in the theoretical literature (e.g., Queller and Strassmann| , |1998| ; |Hughes et al.| , |2008 ; 



Schino et al| |2009| ). 

Whereas early discussions of social evolution usually pitted one mechanism against another, recent ef- 
forts are increasingly focused on synthesizing and integrating the various pathways of social evolution (e.g., 
Marshall and Rowe| , |2003|; |Sachs et al| |2004|; |Lehmann and Keller], |2006a|; |Nowak|, |2006|; |West et al.|, |2007 



Fletcher and Doebeli| , 2009; Q ueller] , |201 1| ). This is perhaps the surest sign that social evolution theory has 



entered a mature phase. These efforts at synthesis are not always without controversy, but they have served 
to illuminate the connections between various models devised over the decades. In this paper, we aim to con- 
tribute to this synthesis by focusing specifically on the relation between three important concepts: assortment 
of genotypes (relatedness), behavioral responsiveness (e.g., reciprocity), and non-additive (e.g., synergistic) ef- 
fects of social behaviors on fitness. These topics have of course been written about in the past; yet, how these 
concepts relate to one another is still not well understood. 

Two of the most fundamental mechanisms for the evolution of cooperation are responsive (and condi- 



tional) behaviors such as reciprocity, and genetic assortment, or relatedness, between individuals (|Lehmann 



and Keller , 2006a). Both factors are clearly in operation in many social interactions. Cooperative breeding, 
for example, almost always involves interactions with relatives of varying degree. At the same time, most 
cooperatively breeding species also exhibit conditional behavior; for example, non-helpers might get evicted 



from the breeding group ( Balshine-Earn et al. , 1998 ). Likewise, reciprocal food-sharing among vampire bats 



can occur amongst relatives as well as non-relatives ( Wilkinson , 1984 ). 

Non-additive interactions between individual phenotypes also play a prominent role in social evolution. 
Sometimes, the non-additivity is a direct function of organismal metabolism, as in the case of the plant 
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pathogen Agrobacterium tumefaciens, which induces infected hosts to produce opines that can be used as a re- 
source only by those cells that also carry the virulence plasmid ([White et al. , 2007 ). In other cases, the 
ecological context might create non-additive payoffs. For example, in the penduline tits (Remiz pendulous), one 
or both of the parents desert the nest after eggs are laid. Data on breeding success suggests that the reason for 



this behavior is in part that a single parent has better than even chance of raising the clutch ( Szentirmai et al. 



2007). Hence, the second parent adds less to breeding success than the first parent, meaning a negative non- 



additivity. It has been argued that the opposite is true for many bird species with biparental care ( jWesolowski , 



1994). It is worth noting that in both of these examples, as perhaps with most instances of non-additive in- 



teractions between phenotypes, the traits in question are plastic. For example, the Agrobacterium pathways 



that induce opine production are not expressed constitutively but depend on quorum-sensing instead ( [White 



;t al.| , |2007| ). Likewise, the male and female decisions to desert in penduline tits appear context-dependent; 
the same individual can care or desert according to factors such as brood size ( Persson and Ohrstrom , 1 989fc 



Valera et alj |1997| ). 

Yet, despite copious theoretical work on the individual importance of conditional behaviors, genetic as- 
sortment, and non-additive payoffs, the interplay between these mechanisms remained relatively unexplored 
until recently (Q ueller , 1985 ; Lehmann and Keller , 2006a ; Fletcher and Zwick , 2006 ; Foster and Wenseleers) , 



20061; Fletcher and Doebelj |2009|; |McGlothlin et al.|, |2010|; Quellerj |201U |Akcay and Van Clevej , |2012| ). 
Much of this recent work neglects to systematically study the interactions between these factors (with the no- 
table exception of Lehmann and Keller , 2006a ; McGlothlin et al. , |2010 ; Akcay and Van Cleve , 2012 ), which 
we show can lead to an incomplete or incorrect view of how selection affects social behaviors. Moreover, 
properly accounting for these interactions allows us to gain novel insights into when conditional behaviors 
should be evolutionarily stable and to empirically assess the level of responsiveness across demographically 
diverse species. 

Our paper begins with two models: first, we use a discrete-action model to clarify the relationship between 
genetic assortment (e.g., relatedness), behavioral responses (a generic notion of reciprocity), and synergistic 
or non-additive interactions. Second, we generalize the results of the discrete-action model by presenting 
a population genetic model for the evolution of a continuous trait in a structured population, where the 
structure can be due to any combination of age, stage, sex, or spatial location. Using these models, we 



show how previous analyses that do not systematically track the effect of behavioral responses ([Fletcher and 
Doebelij |2009fc |Quellerj |201l| ) have neglected a crucial interaction between responsiveness and relatedness 
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( McGlothlin et al. , 2010 ; Akgay and Van Cleve , 2012 ). Rather than combining additively in the equation 
for the effect of selection ( Queller , |201 1 ), relatedness and responsiveness interact through a product term 



that implies each of these assortment mechanisms contributes distinct effects (contra [Fletcher and Doebeli| , 



2009). We also show this crucial interaction to be missing in models of interspecific mutualisms ( poster and 



Wenseleersj , |2006) , which leads to difficulties in interpreting mechanisms that stabilize mutualisms. 



More generally, our discrete model allows us to show how synergistic interactions can affect the evolution- 
ary stability of social behaviors and even change the payoff structure of the game resulting in a different game 



entirely. This is important because helping is more likely to evolve in Hawk-Dove ( Maynard Smith and Price , 



1973 ) or Stag-Hunt ( Skyrms , 2001 ) games than in the Prisoner's Dilemma game. Using our continuous-trait 



model, we show that proximate physiological or psychological mechanisms evolve in the long run only when 
they produce "enough" responsiveness relative to the responsiveness level that exactly balances the demo- 
graphic costs and benefits of the behavior. This evolutionarily stable level of responsiveness is a function of 
both the absolute level of relatedness and the interaction between relatedness and responsiveness. Finally, we 
highlight in the discussion how recent work in quantitative genetics that measures relatedness and respon- 
siveness requires a proper accounting of the interaction of these two mechanisms. 

2. Discrete action model 

We consider a 2x2 game played repeatedly between two individuals a large number of times. We take the 
game to be symmetric for ease of exposition, but this assumption can be relaxed with no consequence for our 
main argument. Label the two actions C and D. Rather than focusing on a particular conditional strategy 



(e.g., the Tit-for-Tat ( |Axelrod and Hamilton! , |1981| )), we model a wide range of propensities to respond to 
one's partner. We assume that each individual i has two traits: a h which is the action (C or D) that the 
individual i intrinsically "prefers", and p t , the probability that individual i overrides its intrinsic preference and 
reciprocates its opponent's action in the last round. Hence, in common parlance, a t corresponds to whether 
individuals are cooperators or defectors, and p t denotes the degree to which individuals are reciprocators. 
Both traits are genetically encoded; we take a { to be binary and p t to be a continuous trait between 0 and 1 . 
A straightforward extension of this model could treat flj as a continuous trait by interpreting it as the intrinsic 
probability of playing C vs. D. 

Consider an interaction between an intrinsic cooperator and an intrinsic defector (i.e., = C and a 2 = 
D). Suppose at stage t, player 1 (the C-type) played C and player 2 (the D-type) also played C (because he 
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was reciprocating cooperation in the previous round). We label this situation as (C, G), where the elements 
correspond to the first and second players' actions, respectively. Let us label the response coefficients of the 
two individuals as p c and p d in according with intrinsic preferences for C and D, respectively. In that case, 
the probability that they will keep playing (C G) at stage t + 1 is the probability that the D-type individual 
keeps reciprocating, i.e., p d . Similarly, the probability that the t + 1 stage will result in (C,D) is 1 - p d , as this 
is the probability that the D-type will revert to his intrinsic preference. Finally, the probabilities of playing 
(A C) and (D,D) at t + 1 given the actions at t is zero, since the C-type player will always play C regardless of 
whether she plays her intrinsic preference or reciprocates. In a similar way, we can work out the probabilities 
of playing any action pair at t + 1 conditional on the action pair at t, which is summarized with the transition 
matrix M = [m„], where m i} gives the probability of being in state j at t + 1 when the game was in state i at t, 
and states are ordered as {(C, G), (C,D), (D, G), (D,D)}. So, mi i = p d , «i 2 = (1 - p d ), etc.. We assume that the 
game is played for a very large number of periods with no discounting, so that the proportion of time each 
pair of actions is played is given by the stationary distribution of the Markov chain M. Taking the dominant 
left-eigenvector of M and normalizing it, we find the stationary distribution of the game between a C-type 
and a Z)-type individual, jt, to be 

x = (a-p c ) 2 p d , (i- Pc )a-Pj),a-P C )P C (i-P d )p d , p c a-P d ) 2 )a-P e P d r 2 ■ (i) 

The expected payoff of a C-type player from playing with a Z)-type player is the fraction of time spent at each 
outcome (given by the elements of n) multiplied by the payoff to her from that outcome. Suppose that the 
stage game is an additive Prisoner's Dilemma: cooperation yields a benefit of b to the partner, at a net cost 
of c to the cooperator. Defection carries no cost and yields no benefit. Thus, the expected payoff to a C-type 
player when playing with a D-type, w CD , is given by 

(1- p c )(bp d -c) 

u>cd = \ ■ (2) 

1 - PcPd 

With a similar calculation, we find the expected payoff to a Z)-type from playing with a C-type to be 

(1 - p c )(b -cp d ) 

w dc = ] ■ (3) 

1 - PcPd 

The expected payoff to a C-type from playing with another C-type is trivial; since both players start with (C, G) 
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and keep playing these actions, w cc = b — c. Similar reasoning yields w DD = 0. 

In order to determine which type of individual is preferred by natural selection, we calculate the condition 
when the frequency of intrinsic cooperators with responsiveness p c increases in a population with intrinsic 
defectors of responsiveness p d . Suppose that some mechanism (e.g., population viscosity, kin-recognition, or 
greenbeards) causes cooperators to assort themselves so that an intrinsic cooperator interacts with another 
intrinsic cooperator with probability r c ; likewise, intrinsic defectors assort among themselves with probability 
r d . When not assorting specifically with their own types, individuals are paired with a partner randomly 
chosen from the population (this is analogous to the classic assumption concerning populations with related 



individuals; see equation 1 in |Rousset and Billia rd, 2000). A full description of the dynamic of frequency 
change in the population would require a specification of the mechanism that generates assortment and 
how it depends on the frequency of genotypes. However, we can leave this unspecified if we only require a 
qualitative picture of frequency change over the course of a single generation. 

Given our assumptions above and that the frequency of the intrinsic cooperative genotype is q, the ex- 
pected fitnesses of an intrinsic cooperator and an intrinsic defector are, respectively, 



W C = (r c + (l- r c )q) w cc + (1 - r c )(l - q) w CD and 
W D = {\- r d )qw DC + (r d + (l- r d )(l - q) w DD . 

The condition for an increase of intrinsic cooperators is W c — W D > 0, which leads to 

b 1 -(1 -q)p c -qp d + (l -q)r c p c {\ - p d ) + qr d (1 - p c ) p d 
c qp c + {\- q)p d + (1 - q)r c (l - p d ) + qr d (l - p c ) - p c p d 



(4) 



(5) 



Three special cases help illustrate the meaning of this equation. First, suppose individuals do not respond to 



each other, p c = p d = 0, and that the assortment probabilities are equal, r c 



r. In this case, condition 



(g| becomes - > -, which of course is Hamilton's rule ( Hamilton , 1964a ). Second, suppose both the intrinsic 
cooperators and defectors have the same responsiveness, p c = p d = p, and assortment is zero, r c = r d = 0. Then, 
condition ^ simplifies to * > i, which is equivalent to other rules for the evolution of responsiveness ( Akcay 



et al.| , |2009| ) and reciprocity (|Lehmann and Keller] , |2006a| ; |Andre and Day| , |2007| ). Finally, giving intrinsic 
cooperators and defectors the same assortment probability and responsiveness, r c = r d = r and p c = p d = p, 
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yields 



b 1 +rp 

- > - 

c r + p 



(6) 



which is analogous to equation (4) in |Lehmann and Keller] fl2006a| ). In condition (g), the assortment coefficient 
r (which is now analogous to relatedness) and responsiveness p have symmetric effects on the increase condition 
but remain distinct components because both the product and sum of p and r appear. Not coincidentally 
expression || is identical in structure to the condition for an evolutionary increase in cooperation we have 



previously derived for continuous public goods games ( [Akcay and Van Cleve| , 12012| ) and to other previously 



derived results ([Lehmann and Keller] , |2006a| ; |McGlothlin et alj , |2010| ). 



It is interesting to contrast our analysis with an earlier model by |Marshall and Rowe| (|2003) that addresses 
a similar problem. In this model, five unlinked loci determine the initial strategy of an individual and the 



responses to each combination of strategies by the two players in the previous round. [Marshall and Rowe 
measure reciprocity with the locus that determines whether an individual retaliates by defecting after it co- 
operated and its opponent defected (a (C,D) outcome in our notation). They find selection for retaliation 



decreases as relatedness increases. Later, |Marshal]| ( j2009| ) finds that this is due to negative non-additivity in 
the payoffs of the Prisoner's Dilemma game they use. In contrast to these models, we model reciprocity as a 
generalized propensity to do what the opponent did in the last round regardless of what the focal individual 
did previously. As such, our results are not readily comparable, since in our model, the four conditional 
responses (one to each potential outcome) are inherently linked to each other. Thus while selection for retali- 
ating after a (C,Z))-round might decrease with increased relatedness in our model (because relatedness favors 
increased cooperation), this decrease is counteracted by increased selection to copy the opponent's behavior 
after a (C, Q outcome. 

3. Population genetic model 

We now turn to a population genetic framework that will allow us to generalize the above results to 
interactions among an arbitrary number of individuals who may be from different demographic classes (e.g., 
males and females, queens and workers, etc.). A population genetic model will also allow us to gain insights 
into how the long-term evolutionary stability of social behaviors depends on the ecological costs and benefits 
and the proximate mechanisms of behavior. At the same time, such generality involves a significant increase 
in the complexity of the model, and hence a different set of simplifying assumptions are necessary. 
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In particular, we are interested in determining when a mutant allele that codes for some phenotype with 
social effects will, on average, increase in frequency in a population and reach fixation. The proper quantity 



to calculate in this case is the fixation probability of the mutant allele ( |Rousset|, |2003| ). For structured popula- 
tions, the fixation probability may be impossible to obtain analytically, so we follow a standard approach that 
approximates that probability using a first-order Taylor-series expansion in <5, which is the difference between 
the mutant and resident phenotypes ( Rousset and Billiard , |2000 ; Rousset and Ronce , |2004 ). Population ge- 
netics theory shows that the first-order term in the expansion is sufficient to calculate the standard conditions 
for the evolutionary stability of phenotypes ( Rousset and Billiard , 2000 ; Rousset and Ronce , |2004 ). More- 
over, this first-order term is proportional to a quantity that is much easier to calculate called the "selection 
gradient", which is the derivative of the expected change in the frequency of the mutant allele with respect to 
8 or dA<//d<5. More intuitively, we can think of Aq « (dA(//d<5) 8 when 8 is small. This means that the sign of 
the selection gradient dAq/ d<5 tells us when the mutant allele is expected to increase or decrease in frequency 
or, in terms of fixation probability, when the mutant allele is more or less likely than a neutral allele to fix in 
the population. 

To obtain the selection gradient, dA<//d<5, we first write a general expression for Aq in a class struc- 
tured model using standard theory in population genetics and evolutionary demography ( Taylor , 1 990fc 
Charlesworth , 1994 ; Caswell , |2001 ; Rousset and Ronce , 2004 ). We do this using an "individually" centered 



approach that was popularized in population genetics by the Price equation ( |Price| , |1972| , |1970| ). The general 



expressions for Aq and dAq/ dS are derived in Appendix [B| in the Supporting Information (equations [B. 1| and 



B.2| , respectively). While general, these expressions do not allow us to understand the interplay between be- 



havioral and demographic mechanisms because we need to specify how the phenotype affects behavior and 
how behavior affects fitness. 

In order to simplify the analysis, we assume a relatively simple social interaction that is unaffected by 
the class structure (though class structure can still affect genetic relatedness). Assume that each individual in 
each class participates in a single social interaction over the course of its lifetime with n — 1 other individuals 
from the same class (e.g., members of a patch interact together). Each individual chooses actions in this social 
interaction (e.g., an amount of effort in some helping behavior). Over the course of the social interaction, 
individuals are assumed to interact with one another many times, and their actions are assumed to reach some 



equilibrium value ([Akcay et al| |2009t |Akcay and Van Cleve| , |2012[) due to a negotiation process jMcNamara 
et al. , 1999 ; Andre and Day , |2007 ; Akcay et al. , |2009 ). This equilibrium action is a* k for individual k and could, 
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for example, measure the amount of effort individual k invests in helping its social partners at the conclusion 
of a negotiation over this amount. If we perturb this equilibrium by changing the action of individual k, we 
can measure the effect of this perturbation on the action of individual m by p km = ^f, which is the response 



coefficient p from section | flAkcay et alj, |2009fc |Akcay and Van Clevej |2012|) 



The negotiation process is driven by some proximate behavioral mechanism whose biological components 
(e.g., neurological structures) are genetically determined (whereas actions themselves are not inherited). For 
simplicity, we assume that a single phenotype/) characterizes the proximate mechanism, and it is the evolution 



of this phenotype that we track with equation ( |B.1[ ). As an example, the phenotype could be a measure of 



hormone function in some brain region associated with cooperative behavior (see |Soares et alj , |2010| , for a 



recent review). The behavioral mechanism influences the negotiation of equilibrium actions, and we can 
measure this effect with which is the effect of individual m's phenotype on the action of individual k. 
This derivative could represent the effect of a change in hormone function in individual m on the amount of 
effort individual k invests in helping. The response coefficient can then be expressed as a function of these 
derivatives: p km = = ^ j ^ • Thus, response coefficients depend on the behavioral mechanism and the 
phenotypes of the individuals involved in the social interaction. 

When the social interaction concludes, we assume that individuals obtain payoffs as a function of their 
equilibrium action and the equilibrium actions of their social partners. We assume this payoff determines 
the fertility of each individual in the interaction (and affects no other fitness component); for example, an 
individual's fertility might be determined by the net calories it obtains from participating in group hunting 
activities where it expends calories when it helps the hunt and it obtains calories when prey are captured by 
itself or its partners. An individual's fertility is denoted by J{p k ) for individual k. We can define the "costs" (c) 
and "benefits" (b) of the social interaction in terms of the affect of actions on fertility where — c = and "•" 
represents the focal individual and b = ^ for some other individual Given these assumptions, we show 
in Appendix [b| in the Supporting Information that the selection gradient dAg/ d<5 is proportional to 

S = bp(n - 1) - c+ r (n - 1) [b (1 + p(n - 2)) - pc] , (7) 

V v ' V w ' 

"direct" effect "indirect" effect 

where p is the response coefficient in a resident population monomorphic for the phenotype p and b and c 



are evaluated at the resident phenotypic value p. This result was previously derived by |Akcay and Van Cleve 



( |2012[ ) and is analogous to expressions found in |McGlothlin et al.| ( |2010[ ). The first term in (|7|) is the "direct 
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effect on the fertility of a focal individual of its own phenotype and actions, whereas the second term is the 
"indirect" effect on the focal individual of the actions and phenotypes of its social partners. It is important 



to note that equation (J7|) is essentially a rearrangement of Hamilton's classic rule ( Hamilton , 1964a ) where 
the effect of local competition due to kin is moved from the benefit and cost terms into the scaled relatedness- 
coefficient r ( Queller , 1994 ; Rousset and Ronce , 2004 ; Rousset, 2004 ; Lehmann and Rousset , 2010| ). We 
derive an expression for r in Appendix [b| in the Supporting Information. 

Assuming that the phenotype p corresponds to investment in a cooperative behavior in a Prisoner's 
Dilemma type of interaction, the condition for an increase in the level of cooperation derived from equa- 
tion (@) is S > 0 or 

b 1 +rp(n- 1) 

c {n - l)(r+ p + (n - 2)rp) ' 

Condition Q is the re-player analog of Even for arbitrarily large interaction groups (re large), the sym- 
metry between relatedness r and responsiveness p is maintained. This reinforces the notion that relatedness 
and responsiveness are distinct causal factors that interact synergistically and cannot be both replaced by a 



single index of "assortment" ( [Fletcher and Doebeli| , |2009| ). Moreover, social interactions with more than two 
individuals (re > 2) introduce an additional product of relatedness and responsiveness; this product is scaled 
by b(n - l)(re - 2) and effectively accounts for the benefit a focal individual obtains when its relatives affect the 
behavior of a third party that interacts with the focal. 

Although we derive equations (0) and || from an explicit population genetic model, the same results can 
be obtained using the tools of path analysis ( Wright , 1920 ; Shipley , |2000 ). Briefly, path analysis in natural 
selection is a multiple regression tool used to measure the causal effect of genetics, environment influences, 
and other variables on fitness. A path analysis diagram connects each causal variable to an effect variable 
via a directed edge. The strength of such effects is measured by the path (or partial regression) coefficients 
that label each edge; unlabeled edges have coefficients that can be neglected since they are common to each 
path. Figure [I] shows the path analysis diagram for the population genetic model described in this section. 

4. Applications 

The discrete action and population genetics models developed above show that an interaction between 
responsiveness and relatedness emerges naturally when considering the effect of selection on a plastic social 
behavior. Below, we illustrate the importance of the interaction between responsiveness and relatedness in 
four ways: first, we show that neglecting the interaction leads to incorrect predictions of the level of coopera- 
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tion through the b/c ratio and argue that past efforts to synthesize mechanisms for cooperation within species 
and for mutualisms between species have neglected this interaction; second, we show how the interaction 
between responsiveness and relatedness can be important in interspecific interactions, such as mutualisms; 
third, we extend the discrete action model to include non-additive or synergistic payoffs and demonstrate 
how responsiveness, relatedness, and synergy interact; and fourth, we use the population genetic model to 
derive a stability condition that reveals how proximate mechanisms evolve in the long run only when they 
generate enough responsiveness relative to the break even point determined by relatedness and the ecological 
costs and benefits of the behavior (i.e. £ = 0). 

4. 1. The interaction between reciprocity and relatedness 

The interaction between responsiveness (i.e., reciprocity) and relatedness in equation (Q) naturally breaks 
into two components: -c(n-\)rp represents the cost to a focal individual of increasing its own level of helping in 
response to the intrinsic increase in the level of helping of related group mates; and b(n- l)(re - 2)rp represents 
the benefit the focal individual receives from third parties that increase their level of helping in response 
to the intrinsic increase in the level of helping of the focal individual's related group mates. Both of these 
components are due essentially to the ability of individuals to respond to the actions of the focal's related 
group mates. Neglecting these responses generates 

- > — (9) 
c r + p 

as the condition for an increase in cooperation. Condition @, which lacks the products of relatedness and 
responsiveness present in condition effectively assumes that related group mates of the focal individual 
do not respond plastically to other individuals whereas unrelated individuals can respond plastically. This 
assumption is very restrictive, which suggests that condition || is incorrect in most cases. 

To visualize how neglecting the interaction between responsiveness and relatedness affects predicted levels 
of cooperation, we plot in Figure |2] the error of the incorrect expression in (^) relative to the correct b/c 
expression in Q as a function of responsiveness. For pairwise interactions (n = 2), the relative error is always 
greater than zero regardless of the value of relatedness, which implies that neglecting the interaction between 
relatedness and responsiveness leads to an overestimate of the level of cooperation. This is the case because 
the neglected interaction consists only of the cost the focal individual pays to respond to its relatives, —c{n—\)rp. 
For larger interaction groups (n > 2), the interaction effect of third parties comes into play b(n — l)(n — 2)rp. 
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This leads to an underestimate of the level of cooperation (relative error less than zero) for small values 
of responsiveness and relatedness and an overestimate for large values of either factor. Thus, in precisely 
the situation where cooperation is expected to be low (both low relatedness and low responsiveness), the 
interaction between responsiveness and relatedness increases cooperation. 

Although an interaction between responsiveness and relatedness is a generic and important component 
of selection on social behavior as we have argued above, it can easily be neglected in efforts to synthesize 
different mechanisms of assortment when those efforts do not systematically track all of the effects of plastic 
responses. Here, we give two examples of such efforts in an attempt to clarify how future syntheses can 
incorporate all the effects of different assortment mechanisms, whether they derive from behavioral plasticity 
within a generation or from demographic mechanisms operating across generations. The first example is an 



expanded Hamilton's rule" presented by |Quellerj(|2011| , equation 1) as the following inequality 



-c+%b i r i +^d i s i + ^m i f i >0, (10) 

% i i 

where c is the cost of a cooperative act to the focal individual, b i is the benefit accruing to the i-th relative 
of the focal individual, r- the relatedness coefficient between the focal individual and the z-th relative. The 
third and fourth terms represent an expansion of the classical Hamilton's rule to include multiplicative effects 
and behavioral feedbacks, which Queller calls "kind" and "kith" selection, respectively. Of these terms, kind 
selection (J], d^ involves a coefficient quantifying the deviation from additivity, d i for the i-th interaction 
partner, which is a "synergistic" fitness effect, and a synergy regression coefficient s v Finally, the last term 
ilLifi m i) 1S me product of a coefficient measuring how the z'-th social partner's phenotype responds to the focal 
individual's phenotype, f h and the fitness effect of this response m ; . Queller suggests that this expansion of 
the phenotypic version of Hamilton's rule represents more types of fitness effects in a causal manner than 
the classical form, and therefore might lead to better disentangling of the various pathways through which 
selection acts on social behaviors. While we agree with the basic premise, we show below that equation (|To| ) 
lacks the interaction between relatedness and responsiveness and is therefore correct only under restrictive 
assumptions. 

To see this, we simplify equation jiot ) by dropping the "kind" selection component (i.e. setting d = 0) and 
obtain 

- c + br + mf > 0 (11) 
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for the case where there is a single class of individuals (i = 1 in equation (10)). In condition ( |1 1[ ), the kith 
selection feedback coefficient,^ is analogous to our behavioral responsiveness measure p. If we let the kith 
fitness effect m be equal to the fitness effect for kin, b, then condition (|TT1 ) immediately becomes condition Q 
that we evaluated above that lacks the interaction between responsiveness and relatedness. The lack of this 
interaction stems from Queller's assumption, without any explicit justification, that the behavioral feedbacks 
and indirect fitness effects can be decomposed additively This assumption is correct only under very re- 
strictive circumstances, namely when only individuals without relatedness to the focal individual respond to 
the focal individual and related individuals do not respond. In Appendix |c] in the Supporting Information, 
we show that adopting this assumption explicitly in Queller's framework leads to condition However, in 
most interactions among social partners with responsive phenotypes, both related and unrelated individuals 
will exhibit behavioral feedbacks, which suggests that a generalized Hamilton's rule that includes both kin 
and kith selection cannot be constructed by simply adding the kin and kith components together. 



Our second example comes from pletcher and Doebeli| (|2009[ ) who argue for the concept of "interaction 



environments" as a way to measure assortment genotype and behavior, regardless of the mechanism that 



generates the assortment. Fletcher and Doebeli argue that the important quantity for determining whether 
cooperators proliferate more than defectors is the difference between the average interaction environment of 
cooperators, e c , and that of defectors, e d , which using our definitions of b and c yields 

b ->^. (12) 
c e c -e d 

The interaction environments depend not only on the genetic assortment probabilities, r c and r d , but also 
on how much each genotype ends up playing C and D against other intrinsic cooperators and defectors, 
respectively. For intrinsic cooperators, e c = r c + ( 1 - r c )(q + (1 - q)(x\ + #3)), where n l is the rth element of the 
stationary distribution of outcomes given in equation (Jl]), so the last term is simply the sum of the fraction of 
games between a C- and Z)-type where the outcome is (C, £) and {D, Q at the stationary distribution. Similarly, 



e d = (I — r d )q(n\ + %<i). Substituting these expressions for e c and e d into condition ( |12[ ) yields 



b > 1 ~ PcPd / 13 v 

c r c {\ - p d )(\ - q) + r d q{\ - p c ) + p d {\ - p c ) + q{p c - p d ) ' 
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which simplifies to 



b 1 +p 

c r + p 



(14) 



when p c = p d = p and r c = r d = r. Comparing this expression with condition (g), we can see that it is not the 
correct condition for an increase in cooperation as it lacks the interaction between relatedness and respon- 
siveness in the numerator. The reason the interaction environment method does not produce the correct 
interaction term rp is because reciprocity and other types of behavioral responses affect a focal genotype's 
fitness not only through changing the social environment a genotype experiences, but also through changing 
the phenotype expressed by the focal genotype in response to this social environment. Therefore, the impli- 
cation that behavioral responses and genetic relatedness are only two special cases of the same fundamental 
assortment quantity does not hold in general since different assortment mechanisms can interact. 

4.2. Reciprocity and synergy 

In the above analyses, and in many biological models of social interactions, the marginal costs and ben- 
efits of cooperation and defection for the focal individual do not depend on the genotype of its partner. 
However, it is often plausible that cooperators might obtain an extra benefit (or cost) from interacting with 
other cooperators, and this benefit (or cost) is called positive (or negative) synergy. The importance of syn- 
ergy, and non-additive interactions more generally, in understanding the evolution of social behavior was 



demonstrated by Queller in his seminal papers (|1984j |1985| ). Since then, many studies have emphasized 
the positive and negative synergies can relax or constrain, respectively, the condition for cooperative behav- 



iors to increase in frequency dQueller] , |1992b| ; |Fletcher and Zwickj , |2006| ; [Hauert et al.[ |2006| ; |Lehmann and 
Kellerj |2006a| fl [Taylor and Nowakj |2007[ phtsukj |2012fc [Taylor and Maciejewskj |2012| ). If positive syn- 



ergies are strong enough, they can also change the structure of the evolutionary game ( |Fletcher and Zwick , 



2006; Hauert et alj , |2006fc [Taylor and Nowak| , |2007| ), which might turn a Prisoner's Dilemma game into a 



Stag-Hunt game. 

Since repeated interactions, such as reciprocity and behavioral responses, in a Prisoner's Dilemma game 



can also stabilize cooperation ( Axelrod and Hamilton , 1981 ; Fudenberg and Maskin , 1986 ), these interactions 



can also be viewed as changing the structure of the one-shot evolutionary game ([Fletcher and Zwick| , |2006 ; 



Lehmann and Keller , 


2006a 


; Andre and Day 


,2007 


; Taylor and Nowak, 


2007; 


Akcay et al. , 


2009; 


Akcay and 



Roughgarden[ |201 \\ . In particular, |Fletcher and Zwick| ( |2006| ) show that reciprocity can alter the evolutionary 



game matrix to produce a game with non-additive payoffs. Going in the other direction, [Marshall and Rowe 
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( 2003 ) and Marshall ( 2009 ) study the evolution of behavioral responses in a particular non-additive game 
matrix. Notwithstanding these few studies, the interaction between behavioral responses and synergy has 
received no systematic study. Here, we show how behavioral responses and synergy have distinct effects 
on the dynamics of social traits and map the changes in the structure of the social game induced by both 
mechanisms simultaneously. 

As is customary in models of synergy, we modify the Prisoner's Dilemma game from Section^ by adding a 
constant d to the payoffs of both individuals when they both cooperate. When d > 0, two cooperators produce 
more benefit together than twice the benefit of a single cooperator; when d < 0, they produce less. These 
cases relate to cooperative acts by the partners being strategic complements versus substitutes, respectively, 



as defined in economics ( [Bulow et al.| , |1985| ). 

Using the same methodology as in section g, we calculate the expected payoff to an intrinsic cooperator 
when paired with an intrinsic defector and an intrinsic defector against an intrinsic cooperator as 



(i - Pc ) {(b Pd -c){\- PcPi ) + d Pd a - Pc )) 

Wq D = ; — and 



a-PcPd) 2 

(i - p e ) {(fi - pjc)( l - pcPd) + dp d a - p c )) 
a-pcpd) 2 



(15) 



respectively. These expressions indicate that reciprocity and payoff synergism can interact in complex ways. 
To better illustrate this interaction, suppose that both types of players have the same probability of recipro- 
cation, p c = p d = p, and the same probability of assortment, r c = r d = r. Then, as shown in Appendix ^ in the 
Supporting Information, the component that the payoff synergy, d, adds to the condition for cooperation to 
increase, W c - W D > 0, is 



d[p + r{\ -p) + 



p{\-2p)(\-r) 
(I+P) 2 



(16) 



Here, the first component, p+r( 1 —p), matches previous work on synergy without reciprocity (|Wenseleers| , |2006 ; 
Lehmann and Keller , |2006b ). The first thing to note from (16) is that, compared with the non-synergistic 



case in condition (g), synergy adds frequency dependence to the increase condition. This is a well-known 
result flGraferj |l979t |Quelleij |l 984|; |Lehmann and Keller], |2006bt |Gardner et alj |2007|; phtsukij, |2012|; ffay- 



lor and Maciejewski , |2012 ) that complicates the analysis of synergy. Essentially, this frequency dependence 



arises from explicitly accounting for the frequency of different discrete pairings, which is necessary when 
payoffs are not additive. Another way of thinking about this frequency dependence is that while relatedness 
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or genetic assortment generally accounts for pairwise genetic correlations, accounting for synergy requires 
calculating triplet correlations, which are generally much more tedious to calculate ([Ohtsuki , 2012 , for ex- 
ample). Notably, this frequency dependence can be eliminated in certain stochastic models with symmetric 



mutation and weak selection (e.g., [Tarnita et al.| , |2009fc [Van Cleve and Lehmanrj , |2013| ) even in the case of 
synergistic payoffs. 

Second, the evolutionary game defined by the expected payoffs from the repeated game, w cc , w CD , w DC , 
and w DD , will take on different game structures depending on the values of rfand p. Figure ^illustrates the dif- 
ferent possibilities. With p < clb, a high enough synergism can transform the game from a Prisoner's Dilemma 
into a Stag-Hunt (coordination) game, which will have two evolutionarily stable strategies (ESSs): all cooper- 
ators or all defectors. If p > 0, then still higher d can turn the game into a Mutualism game, where intrinsic 
cooperators have an unconditional advantage. When p > c/b, the possibility of generating a coordination 
game disappears, and increasing d turns the Prisoner's Dilemma into a Hawk-Dove game (anti-coordination 
or Snowdrift game) and eventually into a Mutualism game as d increases further. To our knowledge, the 
interaction between reciprocity and synergy demonstrated in Figure |§] has not been previously recognized. 
This interaction is particularly important in the light of recent results that show that non-additivity by it- 
self does not change the conditions for the evolution of cooperation in populations with homogenous spatial 



structure (jOhtsuki| , |2012| ), unless it changes the structure of the evolutionary game. 

4.3. The evolution of reciprocity 

While the discrete-action model allows us to derive intuitive relationships between relatedness, reciprocity, 
and synergy, it assumes a specific mechanism of repeated interactions that resembles tit-for-tat or direct reci- 
procity. In contrast, the population genetic model in section ||, where investments in cooperation are con- 
tinuous, can handle families (in the mathematical sense) of flexible behavioral responses. In this section, we 
discuss how to analyze the evolutionary stability of such behavioral response families. 

In our population genetic approach, how individuals act and how responsive they are to their partners' 
actions are both functions of the phenotype (p) describing the proximate behavioral mechanism (see Figure 
U). Mechanistic examples of this phenotype p could include the level of connectivity between particular 
brain regions or the sensitivity of particular neurons to neuromodulators such as oxytocin. Alternatively, 
one can think of p at the functional level where it might represent, for example, the strength of an other- 
regarding preference individuals have towards their social partners ( Akcay et al. , ^0 09). Regardless of whether 



16 



Downloaded from http://biorxiv.org/on September 18, 2014 



the phenotype p is measured at the mechanistic or functional level, its role in our model is to provide a 
mechanism that connects how individuals act to how they respond to the actions of others. Specifically, the 
phenotype (and the behavioral mechanism associated with it) implicitly defines a function p(a) that measures 
the responsiveness level of an individual as a function of the action a of its social partners when those partners 
are monomorphic for the phenotype. For instance, since more intense other-regarding preferences produce 
both higher investments in a group and more responsive individuals, p(a) is an increasing function as depicted 
by the solid black curve in Figure |[ 

Now, we can use the first order condition for evolutionary stability (S = 0 in (Q), to solve for the respon- 
siveness level p that is necessary to make a given action a evolutionarily stable at a given relatedness value r. In 
this way, the first order evolutionary stability condition S = 0 defines another set of curves (one for each value 
of relatedness) that maps actions, a, to responsiveness, p (dashed curves in Figure |J. Then, the intersection 
of the p{a) curve obtained from the proximate mechanism and the curve obtained by setting S = 0 gives the 
evolutionarily stable value of the action and the phenotype that produces it. Note that different phenotypes 
for a single proximate behavioral mechanism or wholly different mechanisms will both in general generate 
different functions p(a) and therefore different evolutionarily stable phenotypes and social behaviors. 

Another important result our population genetic model yields concerns whether populations can evolve 
towards these evolutionarily stable phenotypes when they start from neighboring ones ( Eshel and Motrol , 



1981& [Eshel| , |1983& [Taylor] , [1989)). This kind of evolutionary stability, called convergence stability, is often 



more important in the long-term evolution of continuous phenotypes than the traditional notion of strict 
evolutionary stability ( Wakano and Lehmann , |20 12 ). A candidate phenotypic value is convergence stable 



when the derivative of dA<// d<5 with respect to the resident phenotypic value p is negative ( [Taylor] , [1989J). This 
condition amounts to 



d_ 

dp 



dAq 



dd 



3=0 



da^dpd^ + dbd£ + ^dS_ 
dp \da dp da db da dc 



< 0. 



<y=0 



(17) 



where the derivatives in ( |17[ ) are evaluated at £ = 0 and measure how p, b, and c change due to both direct and 



indirect changes in the phenotype/) and the resulting actions a. Even in this generic form, condition (1_7) tells 
us that the evolutionary stability of a phenotype depends on how a change in the phenotype and the resulting 
action affects the level of responsiveness and how responsiveness changes selection for that phenotype. In 
other words, it is not only the level of responsiveness that matters, but also how the underlying phenotype 
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changes responsiveness and how responsiveness in turn alters the selection gradient. 

More practically, the convergence stability condition can be interpreted graphically as in Figure f§. With 



a little algebra and assuming > 0 (a natural assumption ifp measures a cooperation phenotype and a the 



level of investment in cooperation), condition (|17|) becomes 



dp(a) 



da 



f=0 



<y=o 



da 



where measures the slope of the behavioral mechanism p(a) where it crosses the evolutionary stability 

curve [S = 0) and -^=^ is the slope of the evolutionary stability curve at that same point. In other words, 
condition (|t8| ) implies that, in Figure |], the curve generated by the behavioral mechanism, p{a), must cross 
the evolutionary stability curve, S = 0, from above. Values of p and a on the S = 0 curve can be thought of as 
the actions that result from phenotypes that generate a level of responsiveness that exactly balances the costs 
and benefits of the behavior. Thus, a behavioral mechanism that crosses the S = 0 curve from above is one 
that generates marginally more responsiveness than what is required to exactly balance the costs and benefits, 
and this is what makes it possible for evolution to converge to such a behavioral mechanism. Importantly the 
location of the S = 0 curve will depend on relatedness. For Prisoner's Dilemma type interactions (b(n — 1) > 
c > 0), both dSldr > 0 and dSldp > 0, which means that the S = 0 curve shifts down as relatedness increases. 
In Figure |], we plot two S = 0 curves for hypothetical values of relatedness, r\ < r 2 , and its clear from the 
figure that increasing relatedness increases the convergence stable level of responsiveness for the proximate 
behavioral mechanism given by p(a). 

4.4. Interspecific mutualisms 

In Appendix |5], we illustrate how the methodology of Section 3 and Appendix B in the Supporting Infor- 
mation can be applied to symbiosis (especially mutualistic ones) where host and symbiont phenotypes respond 
to each other and where both species exist in structured populations ( Foster and Wenseleers , ^006 ; Akcay and 



Simmsl , |201 1& |Akcay| , |in reviewj ) . Our results show that both the host and symbiont population structure (char- 
acterized by within- and between-species genetic association coefficients) and phenotypic responses interact 



in previously unrecognized, non-additive ways (compare for example, with [Foster and Wenseleers| , |2006). 
This reinforces our main point, namely that phenotypic responses and genetic associations need to be con- 
sidered in a consistent framework to be accurately integrated. In the interest of space, we refer the reader to 
Appendix [d] in the Supporting Information for the details of the model and the results. 



18 



Downloaded from http://biorxiv.org/on September 18, 2014 



5. Discussion 

One of the major goals of evolutionary and behavioral ecology is to elucidate the causal biological path- 
ways that drive the evolution of social traits. However, natural selection rarely operates through a single 
pathway, and therefore the various causal components of social evolution need to be integrated and their 
commonalities and interactions explored. Our main goal in this paper is to contribute to this synthesis ef- 
fort. In so doing, we highlight that genetic assortment, behavioral responses, and non-additive interactions 
between phenotypes all have both distinct and interacting effects. Therefore, combining these components 
requires explicit and consistent modeling approaches. Using a model based on discrete phenotypes and a 
population-genetic model based on continuous phenotypes, we obtain the following results. 

(i) Phenotypic assortment is not a univariate quantity: it matters how much assortment comes from both 
behavioral responses and genetic assortment. Further, behavioral responses and genetic assortment cannot 
simply be added to each other, even when social interactions (payoffs) are additive. Rather, they interact with 



each other both additively and multiplicatively. Prior results analogous to this can be found in |Lehmann and 
Keller] ( |2006a| ), |McGlothlin et al.| fl2010| ) and |Akcay and Van Cleve| ( |2012| ). Yet, despite this interaction being 



known for some time, its crucial role seems to be not widely appreciated. |Akcay and Van Cleve| (|20 1 2| ) is 
the first paper to our knowledge to highlight the synergistic and symmetric effect of genetic assortment and 
behavioral responses on the response to selection of social phenotypes in social groups of any size. In that 
work, we demonstrated how synergism between genetic assortment and behavioral responses can facilitate 
the transfer of fitness effects from the within-group level to the between-group level, a crucial step in evolu- 



tionary transitions in individuality (|Maynard Smith and Szathmaryl , |1995|; pVIichod| , |2005| ). These previous 
results assumed continuous phenotypes and within-species interactions; here we show that a similar syner- 
gism between genetic assortment and behavioral responses is found for discrete phenotypes and is important 
for species interactions as well. 

(ii) Non-additive or synergistic interactions can have important effects on whether cooperative behaviors 
increase in frequency in a population and whether these effects depend on behavioral responses and reci- 



procity. Specifically, synergy and reciprocity can both change the structure of the evolutionary game ( Taylor 



and Now ak, 2007). When this game starts as a Prisoner's Dilemma where cooperation is unconditional, 
increasing synergy or responsiveness levels can stabilize some level of cooperation. However, whether all in- 
dividuals cooperate fully depends on the combination of synergy and responsiveness that determines whether 
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the game becomes a Stag-Hunt, Mutualism, or Hawk-Dove game (cf. Van Cleve and Lehmann , 2013 ). 

Even though synergistic interactions are easy to study using a discrete phenotypic model, the population- 
genetic model with continuous phenotypes shows the importance of non-additive payoffs or fitness in more 
subtle way. These interactions will affect the evolutionary (convergence) stability of a phenotype and how the 



stable phenotype changes with underlying parameters. The former effect can be seen in equation (|17|), which 
shows that non-additivity in the benefits and costs of a social phenotype (behavior) appear in the convergence 
stability condition (since b and c are themselves already first-order derivatives of fertility). Non-additivity 
in benefits and costs relate directly to ecological mechanisms that convert the effort of individuals in the 
social interaction into payoffs in terms of fertility or survival. For example, individuals who put effort into a 



group hunt ( Skyrms , 2001 ) may receive more return in terms of the amount of prey item per unit of caloric 
investment when more individuals join the hunt. This kind of ecological synergy, or "complementarity" in 



economics flBulow et al| |1985| ), can have large effects on the level of investment in cooperative interactions 



both within groups (|Akcay et al.| , |2009| ) and in structured populations ( |Akcay and Van Cleve| , I2012& |Cornforth 



et al., 2012). 



(iii) There is a simple graphical relationship required for specific proximate behavioral mechanisms (e.g., 
social preferences, social norms, punishment, etc.) to be evolutionarily stable in the long run. When plotting 
the curve of responsiveness as a function of the evolving phenotype for the mechanism (level of other regard, 
strength of conformity, probability of punishment, etc), this curve must cross the curve that balances the costs 
and benefits of the behavior from above. Moreover, since different levels of relatedness result in different 
curves that balance benefit and cost, what level of responsiveness evolves for a specific proximate mechanism 
can also be determined graphically. 

5. 1. Empirical issues 

The empirical study of the interplay between behavioral responses and genetic assortment and the role 
of synergistic payoffs is still relatively new. Most of the recent work linking behavioral responses and genetic 
assortment comes from the application of indirect genetic effect (IGE) models to laboratory, agricultural, and 



field-based populations (e.g. |Moore et al| |2002t |Petfield et al.| , |2005| ; |Mutic and Wolff , |2007| ; |Ellen et al| |2008 



Danielson-Francois et al. 


, 2009; 


Bleakley and Brodie 


, 2009; Teplitskyet al., 


2010; 


Frere et al. , 


2010; 


Wilson 



eTaLj |201 lb |Hamilton and Ligockj |2012| ). IGE models (|Moore et alj , |l997fc |Wolf et al| |1999| ; |McGlothlin 
et al. , |2010 ) extend classical quantitative genetics ( Lynch and Walsh , 1998) to include a coefficient, y/, that 
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measures the effect of a social partner's phenotype on the phenotype of a focal individual. Importantly, 



the IGE coefficient y/ directly maps onto our behavioral responsiveness coefficient p flAkcay and Van Cleve , 



20 1 2| ) and is thus also a measure of reciprocity. We know from equation 0 (and analogously equation 18 



in McGlothlin et al. 2010 ) that an interaction between behavioral responses and relatedness is important 
for selection on social traits, even when no such interaction is assumed to affect individual phenotypes (see 



equation |C.5| in the Supporting Information or equation 5 in ( |McGlothlin et al.| , |2010| )). However, most 
of the empirical work to date spends little time on the interaction of behavioral responses and relatedness 



with the notable exceptions of pleakley and Brodie| (|2009[ ) and |Frere et al.| fl2010| ). Moreover, some empirical 
work assumes an explicit interaction term in the regression model for individual phenotypes, and it is not yet 
clear how empirical estimates of this interaction can be tied to the predicted interaction term that arises in 
measures of selection on the trait. Thus, more work needs to be done understanding the interaction between 
behavioral responses and relatedness both theoretically and empirically. 



Empirical studies explicitly focused on non-additive payoffs are uncommon dQueller et al.| , |2003| ; pore 



et al.| , |2009| , are notable exceptions). This paucity of studies does not reflect the importance of non-additive 
interactions in nature. Instead, non-additive and non-linear interactions are likely to be abundant and their 
effects strong. For example, in parental care, many arguments implicitly or explicitly rely on the contributions 



of each parent combining non-additively (e.g., |Winklerj |1987| ; |Akcay and Roughgarden| , |2009t ). We suggest 
that measuring non-additive interactions at both the payoff (fertility or survival) and fitness levels will prove 
important for explaining the evolution of social behaviors, regardless of whether the phenotypes in question 
vary continuously or are discrete characters with large effects. 

To conclude, a careful analysis of behavioral responses, genetic assortment, and non-additive interactions 
shows a complex interaction between these three pathways. We believe that focusing on this complexity will 
ultimately deliver a more complete and nuanced understanding of the evolutionary forces shaping social 
behaviors. The necessary mathematical framework for such understanding exists, though as we show in 
this paper, it needs to be applied with some care. Once crafted, these theories can be tested against the 
growing abundance of sophisticated datasets that measure both networks of social interactions and important 
demographic and fitness-related variables (e.g. Holekamp et al. , 2012 ; Apicella et al. , |2012 ). 
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Appendix A: Two-player game with synergy 

We follow the discrete- action model defined in section |2[ Individuals are either intrinsic cooperators 
or intrinsic defectors who can either play their intrinsic strategy or reciprocate their partner's last action. 
Let p c and p d be the probabilities that an intrinsic cooperator and defector, respectively, reciprocate or copy 
their partner's last action. We allow for non-random interactions among intrinsic cooperators and intrinsic 
defectors that might result from population structure or other assortation mechanisms and define r c and r d 
to be the probabilities that intrinsic cooperators and defectors, respectively, interact among themselves non- 
randomly. Two cooperators who play each other each earn payoff b-c + d, where b is the benefit, c is the cost 
of cooperation, and d is a synergy term that accounts for non-additive payoffs. A cooperator and defector 
playing together results in a payoff of —c for the cooperator and b for the defector. Two defectors playing 
each other each earn zero payoff. Given this setup and following the analysis of section ^ the payoffs to an 
intrinsic cooperator paired with an intrinsic defector and vice-versa are 

(l-p c ) {(bp d - c)(l - p cPd ) + dp d (l - p c )) 
W CD = 7j ^ and 

(l-p c ) ({b - p d c)(l - p cPd ) + dp d (l - p c )) 

™dc = 71 rn ' 

()-p c PdY 



respectively, which are also given in equation (15) in the main text. 

We can plug the above values for w CD and w DC , along with w cc = b-c + d and w DD = 0, into the expression 
for fitness in equation (Q) to derive the condition for cooperation to increase, W c — W D > 0, which simplifies 
to 

. (PPc + ( 1 - P)Pd + r c ( 1 - P)( 1 - Pd) + r dP( 1 - Pc) - PcPd 

b 



1 - PcPd 

1 - (1 -p)p c -pp d + (l -p)r c p c {\ - p d ) + rjppjil - p c ) 



1 -PcPd ) (A- 1 



+d 



(p + r c (l -p))(l -p d )(\- p]p^ + (1 -p+pr d ){\ - pf Pd 



> 0. 



()-PcPd) 2 

This expression simplifies to condition ^ when d = 0. Another simplification is to assume that the recip- 
rocation and assortment probabilities are the same for cooperators and defectors, namely p c = p d = p and 



r. In this case, expression (jATj) simplifies to 
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which in turn simplifies to condition Q when d = 0. 
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Appendix B: Allele frequency change in a class-structured model 

Suppose that there are classes in the population, where an individual k's class could represent the local 
population it lives in, its age or developmental stage, its sex, or some other phenotypic feature. Each class 
contains JV k individuals and the total population size is N = Yjk-^k- I n or der to track allele frequencies in 
different classes, we let q = (. . . , qj k , . . .) be a column vector of length jV where qj k is the frequency of the allele 
in individual k in class j. Average allele frequencies in each class in the next time period are calculated by 
multiplying a transition matrix W = [«;»] times q, where w^ k is the probability that an allele in a random 
individual in class i descended from individual k in classj. The transition matrix is a function of the fitnesses 
of each class and the transmission processes, such as segregation, mutation, and migration, that affect how 
alleles move between classes. Fitness can include both fertility and survival, and thus populations with both 
overlapping and nonoverlapping generations are possible. We assume that the fitness of individual k in class 
j is function of its phenotype, pj k , and potentially the phenotypes of its social partners. In the simplest cases, 
phenotype depends on genotype in a linear way where p: k = p + qj k S and S is a measure of how much the 
mutant differs from the resident phenotype p. For non-additive genetic interactions (such as dominance or 



epistasis), phenotype depends on genotype in a nonlinear way ( |Gardner et alj , |2007| ). 

Calculating Aq requires us to average the class frequencies in such a way that the average frequency 
remains constant in the absence of natural selection. The appropriate weights for this average are the repro- 



ductive values of each class (^isherj, |1 930fc [Leslie| , |1948| ; [Taylor] , [1990J, a = (... , a k , ...). Thus, the change in 
the reproductive value weighted allele frequency is 



A^ = ct-W-q — ct-N-q 



(B.l) 



where N is a K x N matrix used to create class averages of the allele frequencies q and whose k-th row has 
the value WV k in the columns 1 + £ti Ni to YjL\ an d zero elsewhere for all k. Using equation (B.l), the 
derivative of Aq with respect to 8 is given by 



dAq 
~d5~ 



dW 

dS 



dw 



ijk 



1 = Z Z Z a «-7i7-'//'.'//<" 

ij,l= 1 k= 1 m= 1 



dPlm 



(B.2) 



The components of equation (|B.2[ ) are reproductive value, a h the effect of one individual's phenotype on 



dw„ 



another individual's probability of transmitting an allele from one generation to the next, — — , and the prob- 

Plm 
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ability that those two individuals both have the mutant allele, q ]k q lm . When this last component, q0i m , is 
averaged over all individuals in a class, we obtain a genetic identity probability that can be related to identity 



by descent probabilities from classic population genetics (Ro usset and Billiard! , |2000|; [Rousset and Ronce , 



2004; |Rousset| , |2004j ) that emerge as more common quantities such as F ST ( |Wright| , |1949| ) in simple models 



of population subdivision flRoussetj |2004j ) such as the island model (jWrightj |1931|) . Moreover, these genetic 
identity probabilities correspond to the genetic covariance terms found in regression models in social evo- 



lution ( |Franl4 |1997| ), which when normalized by genetic variance correspond to the regression definition of 



relatedness (|Hamilton| , |1972| ; |Quellerj |1992a[) . 



die 



Recall that the Taylor series is expanded around 8 = 0, so the partial derivatives in (|B.2| ) are evaluated 
at neutrality where natural selection on fertility differences is absent. Additionally, 5 = 0 implies that all 
individuals in a class are equivalent, which means that the partial derivatives above fall into three cases: the 
first case is the effect of a focal individual's phenotype on its own probability of sending an allele to classj in 
the next generation from its own class i, the second is the effect on that probability of another individual's 
phenotype from the same class as the focal, \ and the third is the effect on that probability of the phenotype 

dw- . , 

of an individual in a different class than the focal individual, where / ^ j. Substituting these derivatives 



into (|B.2[ ), we obtain 



dW 

a ■ — — • q : 
d<5 H 



K 



dlV-— dW:: _ dW;; ^ 



d Pj- J ' 



(B.3) 



Further, we assume that the phenotype only affects fertility f (though it could affect survival or demo- 
graphic factors such as carrying capacity or dispersal rates), which means the derivatives in the right-hand 



side of ( p3.3| ) can be split into two components, the effect of the phenotype on fertility and the effect of fertility 



on the probability an allele in class i descends from class j, w~. The latter derivatives we represent with t 



dw,- dw tJ 

where % . = — , % . = ^, 



and 



lJ ' 1 ¥„ 



for / These derivatives essentially contain transmission proba- 



bilities due to dispersal, mutation, allelic segregation, age-dependent mortality, etc, and we assume they are 



34 



Downloaded from http://biorxiv.org/on September 18, 2014 



independent of phenotype. Given these two classes of derivatives, 



din- 



dp 



hn 



= T ... a A + (N .- n T ...ii 



dw- df- 



a/:. a£ 
+ rAr- - 2)— 



for (/, m) = (/', A) 
for / =j and m # £ 
for / #j . 



(B.4) 



Finally, we assume that the effect of phenotype on fertility is independent of the source or destination 
classes of the offspring; that is, we can define -C = -jj- = j- and B = j^- = Combining C and B with 



( |B.4| ) into equation ( |B.3[ ) results in 



dW r 
a ■ — — ■ q = - C 
d<5 H 



B 



1 

A 



=K c (-C + rB) , 



Z - d ( + (%•• + - 2 )^>) + V Z - w - l H 

y=l V / 



(B.5) 



where iT c and K B equal the first and second bracket in (|B.5[ ), respectively. Scaled relatedness is given r = K B IK (: . 



The terms in parentheses on the last line of ( |B.5[ ) are the same as the right-hand side of equation 0, once C 
and B are evaluated following the assumptions in that section that the phenotype p is a parameter of some 
physiological/psychological mechanism that affects the underlying actions individuals choose, a*, and the 
marginal effect of the action of one individual on the action of another is p. A detailed exposition of this 



analysis is given by Akcay and Van Cleve fl2012 ) where -C corresponds to in equation (2) and B to in 



ad- 



equation (3) in that paper. In most biological cases of interest, K c > 0, which results in the proportionality in 



equation 0. The convergence stability condition in ( |17| ) can be derived from equation 0 simply by recalling 
the assumption that the phenotype affects fertility but not demographic quantities contained in r. 
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Appendix C: Uni-directional and bi-directional reciprocity in indirect genetic effects 
models 

In this section, we will show that: (i) assuming a uni-directional type of reciprocity where the focal individ- 
ual can affect its partner's phenotype, but not vice-versa, can lead to the expanded Hamilton's rule presented 



by Queller] ( J20 1 1| , equation 1); and (ii) bi-directional reciprocity leads to condition (|6j), which is equivalent to 



results derived by us ( |Akgay and Van Clevej , |2012| ) and others ( |Lehmann and Keller] , |2006a| ; jMcGlothlin et al. 



20 1 0[ ). Notation in the following equations is taken from Queller] ( J20 1 1[ ) and generally follows quantitative 



genetic approaches to social evolution (e.g., Queller] , |1992b| ; |Frank| , |1997& Queller] , |201 1[ ) with a particular fo 



, 1997; 


Wolf et al., 


1999; 


McGlothlin 



et al., 2010) 



We begin with the phenotypic version of Hamilton's rule, derived from the Price equation (|Price|, |1970[ ), 
that is equation (7) in Queller ( 201 1 ), which 



is 



Pwp-p + Pwp-p 



Cov[G,P] 
Cov[G,P] 



(C.l) 



where we use primes to denote variables associated with the social partner. Suppose that the phenotype of 
the social partner is defined as 

p = G' + fr P P+e', (C.2) 

which says that the phenotype of the social partner is a linear function of its average breeding value (G"), 
the effect of the phenotype of the focal (fip P P), and a random component with mean zero (e', a standard 
assumption in quantitative genetics; see 



individual is given by 



Lynch and Walsh| , |1998|) . Suppose also that the phenotype of focal 
P=G + e, (C.3) 



which says the phenotype of the focal individual is a linear function only of its breeding value and a random 
component; thus, there is no effect of the partner's phenotype on the focal, which embodies the assumption 
we believe underlies Queller (|201 l )'s analysis. Plugging P and P given in equations ( |C.2| ) and (CL3) into 



equation (|C.1| ) yields 



Pwp.p + Pwpp 



Gov [G, G" + PppP+e'] 
Cov[G,G + e] 



Pwp.pi + Pwf P [PpP + Pi 



G'G 



(C.4) 
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which can be written as 



-c + b(p + r) 



in the notation of the main text and is equivalent to the additive combination of kin and kith components in 



equation (1) of |Queller| (|2011 ) 



To include bi-directional reciprocity, we follow the analysis presented by [McGlothlin et al] pOlCf ; [Akcay 



and Van Cleve| fl2012| ) obtain the same results and present a comparison of the relative merits of the current 
approach and the IGE approach. We start by setting 



P=G + /3p P F +e, 



(C.5) 



which expresses the fact that P also has an effect on P. Before we can plug this definition of P or the definition 
of P into equation (7) of jQuelle'r (|201 1 ), we must insert our definition of P into ( C.5 ) and our definition of P 



into flC.2| ) and solve the resulting equations; the solutions are 



P = 



P = 



G+e + Ppp (G' + e') 
1 - Ppp p P , P 

G +e' + Ppp(G+e) 
1 - fop* pp> P 



(C.6) 



These solutions can be compared to the analogous solution in |McGlothlin et al.| (|20 1 0[ ) (equation 6). Plugging 



equations ( |C.6| ) into equation (7) of jQuellerj (|201 \\ yields 



Cov 



Pwp-p + Pwp-p 



q d +e' p (G+e) 



- Pwpp + Pwp P 
= Pwp-p 1 + Pwpp 

= P]\p.p' + &WPP 



Cov [G, G' +e' +t}p, P (G+e)] 
Cov [G,G + e + p P p [G> + e')] 

Cov [G, G'} + 0p, P Var[G] 
Var[G] + fipp Cov [G, G>] 

Pg'g + Ppp 
1 +j)pp,i) c , G 



Thus, the Hamilton's rule one gets from equation (7) of |Quelle"r ( 201 1 ) should be 



, ,„ Pgg + Ppp . n 

PWPP + PWP PTTli a > U 

1 + Ppp'Pc'g 
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or 



( 1 + fippffiffc) Pwp.p + p nr .p {Pgg + h'p) > 0 



(C.7) 



Assuming that fipp = fipp* , equation (|C.7| ) becomes in our notation 



-c( 1 + pr) + b(r + p)>0. 



which is exactly condition Q from our model. This is also exactly equivalent to equation (18) given by 
McGlothlin et al. ( |2010 ) for pairwise interactions once the appropriate mapping between p and their IGE 



coefficient y/ is made (see Appendix A9 in ref |Akcay and Van Cleve| , |2012| ). 
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Appendix D: Interspecific mutualisms 

The interaction between responsiveness and relatedness affects not oniy social behavior between individ- 
uals of the same species but it also shapes social behavior between species. We can illustrate this by applying 
the population genetic model with continuous traits (Section |§] in the main text and Appendix [§]) to an in- 
teraction between a population of host individuals that each interact with a group of re symbionts. Label the 
investment of the host species (H) into the ith symbiont by Xj and the investment of the z'th symbiont (S t ) into 
the host byjy,-. The payoffs (i.e., individual fertility as a result of investments) are denoted u H (x^,--- ,x n ,y\, — ,J>„) 
for the host and for the rth symbiont (i.e., we assume that a symbiont's fertility is only affected by the 

investment it makes into and it receives from the host). 

Suppose further that the investments of both the hosts and symbionts are plastic and can change in 
response to each other. As a result, all of the investment variables are joint functions of the genotypes of 
the individual hosts and symbionts. Following the methodology in Section ^| (see also Akcay et al. , 2009fc 



Akcay and Van Cleve| , |2012| ), we label the host's and symbionts' genetically determined phenotypes p H and 
p s , respectively. Recall that phenotype in this context refers to a physiological mechanism (neurological, bio- 
chemical, etc) that alters investment levels, and that phenotypes are directly genetically determined rather 
investment levels. To find the increase condition for genotypes that result in higher investments in the mu- 



tualism, we apply equation ( |B.2| ) once for each species while treating the partner of the other species as a 
non-reproductive class. 

Calculating the increase condition requires the total derivative of the fertilities (u H and Ug) with respect to 
the host and symbiont phenotypes. For example, the change is host fertility with respect to a change in the 
phenotype of symbiont j is: 



du H ^ du H dx. 



dpsj 



+ y du H dy t 

d*i dpsj i d Ji d Psj 



cxn[-c H + p HS b H ] , 



(D.f) 



where c H = —7^, b H = and p HS = y~Ij^-- The latter quantity denotes the response of a symbiont's 

0 '.. 1 P Sj ' Sj 



investment to the change of the host's investment into that symbiont ([Foster and Wenseleersj , |2006t |Akcay 



and Simms 


,P011; 


Akcay, 


in review 
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(partial) conditions for the ESS phenotypes in hosts and symbionts, respectively, as 



b H 1 + r HS [ps.h. +(n-l )p Sm H ) 
C H p HS + r HS {\+(n-\)p ss ) 
h = 1 + r si-iPHS + (n- \>ssPss 

CS PS H + (« ~ !)W H + r SH ' 



(D.2) 
(D.3) 



where r ss is the relatedness coefficient for the symbionts. The coefficients r HS and r SH measure the genetic 
covariance between higher investment level phenotypes of the symbionts and the host (relative to host and 



symbiont genetic variance, respectively) and are analogous to relatedness coefficients (Frank, 1994 ). The 
conditions above are only partial ESS conditions, since the between species genetic association coefficients 
are dependent on the host and symbiont phenotypes and their fertility effects. To generate the full ESS 
conditions, we would need a set of equations that describe the population dynamics of host and symbionts 
as a function of their investment phenotypes and find the equilibrium values for the genetic associations 



in the structured population, as is done by |Frank| ( |1994| ). Finally, the coefficient p s H denotes how much 
increased investment of a focal symbiont causes the host to increase its investment into the same symbiont, 
p s H measures how the investment in another symbiont by the host responds to a focal symbiont's investment, 
and p S g measures the response of symbionts to each other. 

Similar to the intraspecific condition for the increase of cooperative phenotypes (equation ^), conditions 



( P.2[ ) and ( P-3| ) contain both multiplicative and additive interactions between the different relatedness and 



responsiveness coefficients. Such interactions have not been fully appreciated before (e.g., [Foster and Wense 



feers| , |2006| and have important effects on the level of investments that evolve in the host and symbionts. For 
example, hosts and symbionts that can respond plastically to changes in the investment levels of one another 
result in higher levels of investment by both host and symbiont and this increase is mediated by an interaction 
between relatedness and responsiveness. 
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Figure 1: The path analysis diagram that describes the population genetic model in section |3| where individual 
1 interacts with its n - 1 social partners. Actions (a) are caused by the behavioral mechanism (p), which in 
turn is shaped by a heritable genetic component (additive genetic value, g). The path coefficient r measures 
genetic relatedness between the focal individual and its social partners. In the gray box, solid lines denote the 
effect that an individual's behavioral phenotype has on its own action and these paths have a coefficient of 1 
(by convention). Dashed lines in the gray box have a coefficient of p and represent the effect of an individual's 
phenotype on the actions of its social partners. Actions affect the fertility of the focal individual where the 
focal decreases its own fertility by c and its social partners increase it by b. Finally, fertility is converted to fitness 
through demographic processes. The strength of selection on the phenotype p is calculated by multiplying all 
the coefficients along a path and then summing all paths. This results in equation (0) and shows how products 
of relatedness and responsiveness are created. 
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Figure 2: These panels plot the relative error of the blc expression given in (^) that lacks products of relatedness 
and responsiveness compared to the correct one given in @. Group sizes n are given in each panel. Each 
curve represents a different value of relatedness r where the lower curve has r = 0.05, the middle curve r = 0.5, 
and the upper curve r = 0.95. 
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Probability of reciprocating (responsiveness), p 

Figure 3: The different possibilities for how a Prisoner's Dilemma game with reciprocity and non-additive 
payoffs can get transformed, depending on the probabilities of reciprocation p and the non-additive payoff or 
synergy component d. Pure cooperation is unstable in the Prisoner's Dilemma, and pure defection is unstable 
in the Mutualism game. Both pure cooperation and defection are stable in the Stag- Hunt (or coordination) 
game. Pure cooperation and defection are unstable in the Hawk-Dove game leaving an intermediate level of 
both stable (stable mixed strategy or polymorphism). 
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Figure 4: The graphical relationship between candidate evolutionarily stable response coefficients (p) as a 
function of individuals actions (a) and the response coefficients generated by a hypothetical proximate mech- 
anism that we label p(a). Here, actions might measure the level of helping, and the graphed proximate 
mechanism generates a positive relationship between responsiveness and actions as might be the case for in- 
dividuals with an other-regarding preference for their partner's payoff. Candidate evolutionary stable values 
of p (independent of any proximate mechanism) and a solve S = 0 (see equation ^ and are plotted as dashed 
curves for hypothetical values of b, c, and n and for two hypothetical values of relatedness, T\ < r 2 . The prox- 
imate mechanism above generates convergence stable actions, at and at, since its response coefficient curve 
M crosses the S = 0 curves from above (see equation @. ' " 



44 



